This invention relates to the field of monitoring computer usage generally, and specifically to software applications for detecting system abuse. Many web site operators provide services to a group of users. Some web sites, referred to as portals, integrate multiple services together to present its users with a comprehensive online experience. Examples of the services provided by web sites and portals include search engines, instant messaging, e-mail, message boards, news, financial data, sports, and online shopping and auctions.
Many web site operators and especially portal operators provide these services for free or at low cost, often solely for their users' personal use. However, unscrupulous users and organizations seek to misuse or overuse these for financial gain or nefarious purposes. Service misuse, overuse, and other wrongful exploitation of a web site's services is referred to generally as “abuse.”
Examples of abuse include crawling a web site to steal and repackage its content, sending spam or unsolicited advertising messages via e-mail, message boards, or instant messaging, and requesting a large number of search requests to find recently expired Internet domain names that can be purchased and ransomed. Abuses cost the web site operator money, for example in the form of extra computing resources needed to handle abuse traffic, lost revenue from legitimate users driven away by the abuse, and increased fees paid by the web site operator to service providers, such as search engines, for the services consumed by abusers.
Preventing abuse is a difficult challenge. First, there is the problem of detecting abuse. A web site service may have thousands or millions of simultaneous users. Because of the large number of users, it is difficult for current monitoring systems to detect abuse as it occurs. Instead, most monitoring systems review service log files sporadically, for example daily, to determine if abuse has previously occurred. If a service log reveals an abusive user, that user can be blocked from the service. However, by the time the abuse is detected, the abuser has already changed identities, such as IP addresses or user IDs, making it difficult to prevent future abuse. Furthermore, processing service logs is a time-consuming task.
Additionally, abusers can take steps to mask their abusive behavior. For example, if a service is provided by multiple servers, the abuser can distribute their service requests among different servers, thereby diluting the number of service requests in each service log. Unless the web site operator cross-references all of its service logs, which greatly increases the time and resources spent processing service logs, this abuse will go undetected.
Moreover, monitoring software must be careful to only block service abusers and not legitimate users. Because of the large number of users that must be analyzed in service logs, it is often too expensive in terms of time and resources to apply complex analyses of user activity, referred to as filters, to identify abusers. Instead, monitoring software must rely on simplistic filters that are either over-inclusive, thereby blocking legitimate users, or under-inclusive, thereby allowing service abusers.
It is therefore desirable for monitoring system to detect service abuse as it occurs, rather than in subsequent analysis. It is further desirable that the monitoring system detect service abuse spread over multiple services. It is still further desirable that the monitoring system allow for complex filters to effectively screen out abusers while allowing for legitimate users.